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ABSTRACT 


This  thesis  investigates  the  properties  of  a  software  package  called  HNeT 
(Holographic/Quantum  Neural  Technology)  which  is  based  on  the  use  of  an  artificial 
intelligence  tool  called  Neural  Networks.  The  basis  for  the  investigation  of  this  software 
is  to  establish  its  reliability,  effectiveness  and  efficiency.  Neural  technology  is  a 
technological  replication  of  the  biological  neural  system  designed  to  learn  data  patterns 
and  process  the  data  (stimulus)  and  then  generate  a  response  based  on  the  memory  of  the 
data.  HNeT  theory  is  fundamentally  different  from  the  standard  Artificial  Neural  System 
(ANS)  in  that  it  uses  complex  scalars  to  evaluate  internal  mappings  of  one  set  of  values 
(stimuli)  to  another  set  of  values  (responses).  HNeT  employs  a  process  known  as 
enfolding,  which  allows  the  learning  and  subsequent  recall  of  many  stimulus-response 
associations  to  be  compressed  into  a  single  HNeT  neuron  cell  improving  the  speed  of 
learning  and  recall  accuracy  as  well  as  reducing  storage  requirements.  Whereas  the 
traditional  ANS  stores  stimulus  patterns  separately  as  a  reference  template  within  a  cell 
and  are  compared  one  at  a  time  to  a  new  incoming  stimulus  response  pattern  which  in  this 
case,  requires  larger  amounts  of  memory. 
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I. 


INTRODUCTION 


The  ability  of  machines  to  perform  accurate  function  prediction  remains  an 
unsolved  problem.  Although  conventional  sensors  used  in  military  applications  provide 
enough  information  for  a  human  to  predict  an  events  outcome,  the  extension  to  automatic 
prediction  by  machines  is  still  impractical  using  current  computer  designs  and  methods  of 
construction. 

HNeT  is  a  program  that  is  designed  to  provide  alternative  methods  for  predictions 
of  outcomes  by  way  of  artificial  neural  networks.  Testing  this  software  will  provide 
information  on  its  reliability,  efficiency  and  whether  it  will  provide  an  adequate  vehicle  of 
improvement  for  military  applications.  For  instance,  if  a  SCUD  missile  is  fired,  will  this 
technology  be  instrumental  in  air  defense  detection  through  image  processing  or 
similarly,  if  a  tank  is  positioned  on  the  battlefield,  will  this  technology  enhance  the 
probability  of  its  detection. 

These  applications  will  not  be  tested  directly,  however,  from  examining  the 
software  we  will  be  able  to  provide  valuable  information  that  will  lead  to  potential  further 
investigation  of  the  previously  mentioned  instances. 

A.  BACKGROUND 

Biologists  have  studied  the  human  brain  for  many  years.  The  better  we  understand 
the  brain,  the  better  we  can  emulate  it,  and  build  artificial  “thinking  machines”  that 
process  and  respond  just  as  the  human  brain.  As  information  about  the  functions  of  the 
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human  brain  accumulated,  new  technologies  arose  and  the  search  for  an  artificial  neural 
network  began. 

Holographic  Quantum  Neural  Technology  (HNeT)  is  a  software  package  that  is 
modeled  to  operate  in  the  fashion  of  a  human  brain.  It  supports  a  framework  in  which  the 
operation  of  stimulus-response  learning  and  recall  may  be  performed  within  single  neuron 
cell  structures.  The  HNeT  library  houses  seven  neuron  cell  types,  four  of  which  are 
modeled  after  the  cerebellum  and  neo-cortex  (the  more  predominate  cells).  The 
predominate  cells  are  the  granulate,  stellate,  pyramidal,  and  the  purkinjee  cells. 

The  mathematical  concepts  in  HNeT  are  somewhat  abstract,  however,  you  do  not 
need  a  full  understanding  of  the  theory  to  run  applications.  It  is  important  that  you 
understand  how  stimulus-response  information  is  presented  to  the  system,  and  how  the 
various  types  of  holographic/quantum  neural  cells  interact  with  each  other. 

The  number  of  scalars  within  that  matrix  can  be  no  larger  than  the  number  of  scalars 
within  only  one  stimulus  pattern. 

B.  OBJECTIVES 

The  objectives  for  this  thesis  are  to  determine  the  differences  between  traditional 
artificial  neural  networks  (ANNs)  and  HNeT  and  to  evaluate  the  software’s  response  to 
two  mathematical  concepts.  The  first  is  simple  function  evaluation.  It  will  be  determined 
if  the  software  is  capable  of  learning  to  emulate  various  simple  functions.  The  reason  for 
starting  with  these  functions  is  to  test  HNeT’s  capability  of  generating  response-recall  of 
common  functions  that  is  familiar  to  most.  After  HNeT  is  evaluated  using  these 
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functions,  the  error  of  the  output  will  give  adequate  information  on  this  software’s 
capability  of  evaluating  simple  functions. 

The  second  concept  is  evaluating  the  random  number  phenomenon  provided  by 
California  Lotto  data.  This  is  a  more  sophisticated  operation  in  that  HNeT  will  be 
required  to  guess  a  response  given  a  set  of  values  (stimulus)  from  randomly  generated 
numbers.  There  will  be  several  versions  of  testing  of  this  concept  to  verify  the  software’s 
reliability.  The  objective  is  to  determine  if  HNeT  can  discern  the  statistically  meaningful 
content  of  the  stimulus. 
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II.  NEURAL  NETWORKS  AND  HNET:  A  COMPARISON 


A.  WHAT  IS  A  NEURAL  NET? 

1.  Artificial  Neural  Networks 

An  artificial  neural  network  is  a  system  for  information  processing  whose 
performance  characteristics  are  analogous  to  biological  neural  networks.  Artificial  neural 
networks  have  been  developed  as  generalizations  of  mathematical  models  of  neural 
biology,  based  on  assumptions  that  information  processing  occurs  at  many  simple  neuron 
elements,  connecting  links  are  used  to  pass  signals  between  neurons,  each  connecting  link 
has  an  associated  weight  which  attenuates  the  signal  transmitted  in  a  typical  neural  net, 
and  each  neuron  applies  an  activation  function  (nonlinear  in  most  cases)  to  its  net  input  to 
determine  its  output  signal. 

Characterizations  of  a  neural  network  are  (1)  its  pattern  of  connections  between 
the  neurons  ( architecture ),  (2)  its  method  of  determining  the  weights  on  the  connections 
which  is  called  its  training  or  learning  algorithm,  and  (3)  its  activation  function. 

A  large  number  of  simple  processing  elements  called  neurons,  units,  cells,  or 
nodes  are  the  primary  makeup  of  a  neural  net.  Each  neuron,  with  an  associated  weight,  is 
connected  to  other  neurons  by  means  of  directed  communication  links.  The  weights 
represent  information  being  used  by  the  net  to  solve  a  problem.  Neural  nets  can  be 
applied  to  a  wide  variety  of  problems,  such  as  storing  and  recalling  data  or  patterns, 
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classifying  patterns,  performing  general  mappings  from  input  patterns  to  output  patterns, 
grouping  similar  patterns,  or  finding  solutions  to  constrained  optimization  problems. 

Each  neuron  has  an  internal  state  which  is  a  function  of  the  inputs  received.  These 
internal  states  are  called  its  activation  or  activity  level.  A  neuron  typically  sends  its 
activation  one  signal  at  a  time  to  several  other  neurons. 

Fausett  [Ref  1]  provides  the  following  example  of  a  neuron  activation.  Consider  a 
neuron  Y,  illustrated  in  Figure  1,  that  receives  inputs  from  neurons  Xj,  X2,  and  X3.  The 
activations  (output  signals)  of  these  neurons  are  xh  x2,  and  x3,  respectively.  The  weights 
on  the  connections  from  Xj,  X2,  and  X3  to  neuron  Y  are  wh  w2,  and  w3,  respectively.  The 
net  inputs,  yjn,  to  neuron  Y  is  the  sum  of  the  weighted  signals  from  neurons  X,,  X2,  and 

X3,  i.e., 

y_in  =  wj  xj  +  W2  *2+  w3  x3  = 

1=1 

The  activation  y  of  the  neuron  Y  is  given  by  some  function  of  its  input  ,y=f(y_in),  e.g., 
the  logistic  sigmoid  function  (an  S-shaped  curve) 

fM  = _ I -  (2 

H  }  1  +  exp(-x) 


6 


or  any  of  a  number  of  other  activation  functions.  Now  suppose  further  that  the  neuron  Y 
is  connected  to  neurons  Zi  and  Z2,  with  weights  vy  and  v?,  respectively,  as  shown  in 
Figure  2. 


Neuron  Y  sends  its  signal  y  to  each  of  these  units.  However,  in  general ,  the  values 
received  by  neurons  Zj  and  Z2  will  be  different,  because  each  signal  is  scaled  by  the 
appropriate  weight,  vy  or  v^.  In  a  typical  net,  the  activations  zy  and  z?  of  neurons  Zj  and  Z2 
would  depend  on  inputs  from  several  or  even  many  neurons,  not  just  one,  as  shown  in  this 
simple  example. 

Although  the  neural  network  in  Figure  2  is  very  simple,  the  presence  of  a  hidden 
unit,  together  with  a  nonlinear  activation  function,  gives  it  the  ability  to  solve  many  more 
problems  than  can  be  solved  by  a  net  with  only  input  and  output  units.  On  the  other  hand, 
it  is  more  difficult  to  train  (i.e.,  find  optimal  values  for  the  weights)  a  net  with  hidden 
units. 


7 


2.  Biological  Neural  Networks 

The  extent  to  which  an  artificial  neural  network  models  a  particular  biological 
neural  system  varies,  which  causes  much  concern  for  some  researchers.  For  others,  the 
ability  of  the  net  to  perform  useful  tasks  is  the  focal  point  of  continued  research  rather 
than  the  biological  plausibility  of  the  net. 

There  is  a  close  analogous  relationship  between  a  biological  neuron  (brain  or 
nerve  cell)  and  an  artificial  neuron  (processing  element).  There  are  three  components  of  a 
biological  neuron  that  are  of  particular  interest  in  understanding  an  artificial  neuron. 
Those  components  are  dendrites,  soma,  and  axon.  The  dendrites  (many  in  number) 
receive  signals  from  other  neurons.  The  signals  that  are  transmitted  are  electric  impulses 
that  travel  by  means  of  a  chemical  process  across  a  synaptic  gap.  The  action  of  the 
chemical  transmitter  modifies  the  incoming  signal  (typically,  by  scaling  the  frequency  of 
the  signals  that  are  received)  in  a  manner  similar  to  the  action  of  the  weights  in  an 
artificial  neural  network. 

The  cell  body,  known  as  the  soma,  sums  the  incoming  signals.  When  sufficient 
input  is  received,  the  cell  transmits  a  signal  over  its  axons  to  other  cells.  The  frequency  of 
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transmitting  varies  and  can  be  viewed  as  a  signal  of  either  greater  or  lesser  magnitude. 
This  process  is  closely  matched  to  looking  at  discrete  time  steps  and  summing  all  activity 
(signals  received  or  sent)  at  a  particular  point  in  time. 

The  transmission  of  the  signal  from  a  particular  neuron  is  accomplished  by  an 
action  potential  resulting  from  distinctive  concentrations  of  ions  on  either  side  of  the 
neuron’s  axon  sheath  (the  brain’s  “white  matter”).  Potassium,  sodium,  and  chloride  ions 
are  most  directly  involved. 

Figure  3  shows  a  generic  illustration  of  a  biological  neuron,  together  with  axons 
from  two  other  neurons  (from  which  the  illustrated  neuron  could  receive  signals)  and 
dendrites  for  two  other  neurons  (to  which  the  original  neuron  would  send  signals). 


Axon  from 


Dendrite  of 
Another  Neuron 


Dendrite  of 
Another  Neuron 


Figure  3.  Biological  Neuron.  From  Ref.[l]. 


Fausett  [Ref.  1]  gives  several  key  features  of  the  processing  elements  of  artificial 


neural  networks  suggested  by  the  properties  of  biological  neurons,  viz.,  that: 

1 .  The  process  element  receives  many  signals. 

2.  Signals  may  be  modified  by  a  weight  at  the  receiving  synapse. 

3.  The  process  element  sums  the  weighted  input. 

4.  Under  appropriate  circumstances,  (sufficient  input),  the  neuron  transmits  a 
signal  output. 
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5.  The  output  from  a  particular  neuron  may  go  to  many  other  neurons  (the  axon 
branches). 

6.  Information  processing  is  local  (although  other  means  of  transmission,  such  as 
the  action  hormones,  may  suggest  means  of  overall  process  control). 

7.  Memory  is  distributed: 

a.  Long  term  memory  resides  in  the  neurons’  synapses  or  weights. 

b.  Short  term  memory  corresponds  to  the  signals  sent  by  the  neurons. 

8  A  synapse’s  strength  may  be  modified  by  experience. 

9.  Neurotransmitters  for  synapses  may  be  excitatory  or  inhibitory. 


Artificial  neural  networks  share  another  important  characteristic  with  biological 
neural  systems  which  is  called  fault  tolerance.  Biological  systems  are  fault  tolerant  in  two 
respects.  First,  we  are  able  to  recognize  many  input  signals  that  are  somewhat  different 
from  any  signal  we  have  seen  before.  An  example  of  this  is  our  ability  to  look  at  a  picture 
we  have  not  seen  before  and  recognize  a  person  in  that  picture  or  to  recognize  a  person 
after  a  long  period  of  time. 

Second,  we  are  able  to  tolerate  damage  to  the  neural  system  itself.  Johnson  & 
Brown  [  Ref.  2  ]  state  that  humans  are  bom  with  as  many  as  100  billion  neurons.  Most  of 
these  are  in  the  brain,  and  most  are  not  replaced  when  they  die.  In  spite  of  our  continuous 
loss  of  neurons,  we  continue  to  leant. 

3.  Adaline  and  Madaline 

We  will  begin  this  section  by  first  introducing  some  activation  functions  most 
common  to  artificial  neural  networks.  An  activation  function  is  defined  as  a  function  that 
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transforms  the  net  input  to  a  neuron  into  its  activation.  It  is  also  known  as  a  transfer,  or 
output  function. 

The  first  function  to  be  discussed  was  introduced  previously.  It  is  known  as  the 
binary  sigmoid  or  the  logistic  sigmoid. 


f(x)  = - - - . 

1  +  exp(-<zc) 

(3) 

f’(x)  =qf(x)  [1  -f(x)}. 

(4) 

This  function  is  illustrated  in  Figure  4  for  two  values  of  the  steepness  parameter 
a.  This  function  is  used  for  neural  nets  in  which  the  desired  output  values  either  are 
binary  or  are  in  the  interval  between  zero  and  one.  It  is  important  to  point  out  that  as  a 
approaches  infinity,  the  function  becomes  a  binary  step  function. 


Figure  4.  Binary  Sigmoid  Steepness  Parameter  cr  =1  and  cr  =  3.  From  Ref.  [  1  ]. 


The  bipolar  sigmoid  is  another  activation  function  that  is  very  common  to  neural 
nets.  This  function  is  often  used  as  an  activation  function  when  the  desired  range  of 
output  values  is  between  minus  one  and  one.  It  is  illustrated  in  Figure  5  for  cr  =  1. 
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The  bipolar  activation  function  is  the  function  most  commonly  used  with  the  Adaline 
device.  Adaline  was  originally  conceived  by  Widrow  and  Hoff  initially  called  the 
ADAptive  Linear  NEuron  but  later  became  the  ADAptive  LINear  Element.  It  is  a  feed 
forward  net  consisting  of  a  single  processing  element  (neuron).  It  receives  input  from 
several  units.  Figure  6  depicts  the  Adaline  structure.  It  is  almost  identical  to  the  simple 
(artificial)  neuron  previously  described,  but  it  has  two  modifications  that  make  it  the 
Adaline.  The  first  is  the  addition  of  a  connection  with  weight,  w0  which  is  called  the  bias 
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term.  This  term  is  a  weight  on  a  connection  whose  input  value  is  always  equal  to  one.  The 
second  modification  is  a  bipolar  activation  function  on  the  output. 


Figure  6.  Adeline  Structure.  From  Ref.  [3  ]. 

There  is  a  part  of  the  Adaline  called  the  adaptive  linear  combiner  (ALC).  It  is 
pictured  in  Figure  6.  If  the  output  of  the  ALC  is  positive,  the  Adaline  output  is  positive 
one.  If  the  ALC  output  is  negative,  the  Adaline  output  is  negative  one.  The  process  done 
by  the  ALC  is  very  similar  to  that  of  the  processing  element  previously  discussed  in  the 
first  section.  It  produces  a  sum-of-products  calculation  using  the  input  and  weight 
vectors,  and  applies  an  output  function  to  get  a  single  output  value  thus  giving  us  the 
following  equation. 


y  =  wo  +  YjWjxj  (?) 

7  =  1 

In  this  equation,  w0  is  the  bias  weight.  If  we  set  xq  =  1,  then  we  rewrite  the  previous 
equation  as 
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or  its  corresponding  vector  notation. 


y  =  wTx.  (9) 

In  this  particular  case,  the  output  is  the  identity  function  as  well  as  the  activation  function. 
This  means  that  the  output  is  the  same  as  the  activation,  which  is  the  same  as  the  net  input 
to  the  unit. 

The  Adaline  can  be  trained  using  the  least  mean  squares  (LMS)  rule  also  known 
as  the  delta  rule.  It  is  a  method  used  in  finding  the  desired  weight  vector.  This  rule 
minimizes  the  mean  squared  error  between  the  activation  and  the  target  value.  Because  of 
this  rule,  the  net  can  continue  learning  on  all  training  patterns.  It  is  mathematically 
described  bellow  as: 


w(r  +  l)=  yv(t)+2juekxk  (10) 

where:  p  =  the  learning  rate 
8k  =  error  value 
Xk=  input  vector. 


If  there  is  an  instance  where  Adalines  are  combined  so  that  outputs  from  some  of 
them  become  inputs  for  others  of  them,  then  the  net  becomes  multilayer  or  many 
Adalines.  This  net  is  known  as  Many  ADAptive  Linear  NEurons  or  Madaline.  Figure  7 
below  shows  the  structure  of  Madaline. 


Figure  7.  Madaline  Structure.  From  Ref.  [  3  ]. 


There  are  three  basic  training  algorithms  for  Madaline  of  which  two  will  be 
discussed.  The  first  is  MADALINE  RULE  I  (MRI).  This  algorithm  was  made  such  that 
only  the  weights  for  the  hidden  Adalines  were  adjusted.  The  second  is  MADALINE 
RULE  II  (MRII).  This  algorithm  provides  a  method  for  adjusting  all  weights  in  the  net. 
The  aim  of  this  algorithm  is  to  cause  as  little  disturbance  as  possible  to  the  net  at  any  step 
of  the  learning  process,  in  order  to  cause  as  little  “unlearning”  of  patterns  for  which  the 
net  had  been  previously  trained.  Freeman  &  Skapura  [Ref.  3  ]  embodied  this  principle  in 
the  following  algorithm: 


1 .  Apply  a  training  vector  to  the  inputs  of  the  Madaline  and  propagate  it  through 
the  output  units. 

2.  Count  the  number  of  incorrect  values  in  the  output  layer;  call  this  number  the 
error. 

3.  For  all  units  on  the  outer  layer, 

a.  Select  the  first  previously  unselected  node  whose  analog  output  is  closest  to 
zero.  (This  node  is  the  node  that  can  reverse  its  bipolar  output  with  the  least 
change  in  its  weights-hence  the  term  minimum  disturbance.) 
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b.  Change  the  weights  on  the  selected  unit  such  that  the  bipolar  output  of  the 
unit  changes. 

c.  Propagate  the  input  vector  forward  from  the  inputs  to  the  outputs  once 
again. 

d.  If  the  weight  change  results  in  a  reduction  in  the  number  of  errors,  accept 
the  weight  change;  otherwise,  restore  the  original  weights. 

4.  Repeat  step  3  for  all  layers  except  the  input  layer. 

5.  For  all  units  on  the  outer  layer, 

a.  Select  the  previously  unselected  pair  of  units  whose  analog  outputs  are 
closest  to  zero. 

b.  Apply  a  weight  correction  to  both  units,  in  order  to  change  the  bipolar 
output  of  each. 

c.  Propagate  the  input  vector  forward  from  the  inputs  to  the  outputs. 

d.  If  the  weight  change  results  in  a  reduction  in  the  number  of  errors,  accept 
the  weight  change;  otherwise,  restore  the  original  weights. 

6.  Repeat  step  5  for  all  layers  except  the  input  layer. 


Steps  5  and  6  are  a  modification  of  the  algorithm  attempting  to  modify  pairs  of 
units  at  the  first  layer  after  all  of  the  individual  modifications  have  been  attempted.  If 
necessary,  the  sequence  can  be  repeated  with  triplets  of  units,  or  quadruplets  of  units,  or 
even  larger  combinations,  until  satisfactory  results  are  obtained. 


The  next  rule  that  will  be  discussed  is  called  the  GENERALIZED  DELTA 
RULE  (GDR).  This  rule  is  used  when  performing  back  propagation  .  This  network 
addresses  problems  requiring  recognition  of  complex  patterns  and  performing  nontrivial 
mapping  functions.  Freeman  &  Skapura  [Ref.  3  ]  states  that  the  network  is  embodied  by 
the  following  description: 


1 .  Apply  an  input  vector  to  the  network  and  calculate  the  corresponding  output 
values. 

2.  Compare  the  actual  outputs  with  the  correct  outputs  and  determine  a  measure 
of  the  error. 
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3.  Determine  in  which  direction  (+  or  -)  to  change  each  weight  in  order  to  reduce 
error. 

4.  Determine  the  amount  by  which  to  change  each  weight. 

5.  Apply  the  corrections  to  the  weights. 

6.  Repeat  items  one  through  five  with  all  the  training  vectors  until  the  error  for 
all  vectors  in  the  training  set  is  reduced  to  an  acceptable  value. 


B.  HNET 


1.  Mathematical  Concepts 

AND  [Ref.  4]  states  that  HNeT  theory  is  fundamentally  different  from  the 
standard  already  discussed  Artificial  Neural  System  (ANS)  theory.  The  neuron  cell  within 
the  holographic/quantum  neural  model  follows  a  non-connectionist  model,  which  implies 
that  learning  and  subsequent  recall  of  stimulus-response  associations  are  performed 
within  single  neuron  cells.  This  is  simply  saying  that  the  operational  features  exhibited  by 
connectionist  neural  networks  can  be  compressed  into  a  single  HNeT  neuron  cell  thus 
improving  the  speed  of  learning  and  recall  accuracy  for  one  HNeT  cell  over  traditional 
connectionist  models. 

The  stimulus  and  response  information  or  patterns  may  be  represented  by  a  set  of 
values,  each  value  residing  within  some  analog  range.  These  sets  of  values  represent  data 
values  measured  within  an  external  environment  with  conditions  such  as  pressure, 
brightness,  temperature,  etc.  During  stimulus-response  learning,  neural  cells  “map”  one 
set  of  values  (stimulus)  to  another  set  of  values  (responses).  The  neural  cell  subsequently 
operates  in  a  manner  that  by  exposing  the  cell  to  the  stimulus,  generation  of  the  second 
field  is  produced  (i.e.  a  response  recall). 
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The  mathematical  basis  for  HNeT  permits  such  stimulus-response  patterns  to  be 
learned  or  “mapped”  within  a  single  matrix  comprised  of  complex  or  real  valued  scalars. 
The  number  of  scalars  within  that  matrix  can  be  no  larger  than  the  number  of  scalars 
within  only  one  stimulus  pattern. 

The  following  linear  model  (11)  using  summation  as  well  as  inner  product 
notation  is  an  example  of  the  operation  that  takes  place  within  the  holographic/quantum 
neuron  cell  where  Xn  is  the  cortical  memory  element. 


(ID 


=  ( Sn  |  R )  integrated  over  time  (t) 

where  in  this  case  Sn  =  \fntee"-‘  =  {y,^' } 

X  —  the  assigned  confidence  level  for  the  stimulus 
y  =  the  assigned  confidence  level  for  the  response 
9  =  phase  orientation  for  the  stimulus 
(|>  =  phase  orientation  for  the  response 


Once  the  initial  stimulus-response  has  cycled  through,  the  above  operation  superimposes 
or  enfolds  information  pertaining  to  all  of  the  stimulus-response  patterns  onto  one  scalar 
which  is  stored  by  the  cortical  memory  element  Xn.  This  process  is  characterized  by  the 
following  equation  (12)  which  introduces  the  response  recall  operation.  In  this  case,  the 
form  is  similar, 


R '  =  ClfS*  |  X)  summed  over  the  cortical  memory  element  (n)  (12) 
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however,  the  inner  product  is  formed  over  the  cortical  memory  elements  (indexed  by  n) 
instead  of  time  and  R '  is  the  response  generated  by  HNeT.  A  new  stimulus  S  is 
transformed  through  all  of  the  stimulus-response  memories  enfolded  within  the  cell’s 
cortical  memory  elements  and  normalized  by  Qs  which  is  generally  some  function  of  the 
stimulus  field.  This  process  just  described  is  referred  to  as  linear  search  which  is  one 
approach  to  pattern  recognition. 

In  the  more  standard  approach  to  pattern  recognition  (i.e.  artificial  neural 
networks),  stimulus  patterns  “memory”  are  stored  separately  as  a  reference  template. 
These  reference  templates  are  compared  one  at  a  time  to  a  new  incoming  stimulus  pattern 
during  a  response  recall  operation  as  we  saw  in  the  previous  section.  From  this  fact,  AND 
[Ref.  4]  implies  that  this  method  would  require  large  amounts  of  memory,  is 
computationally  intensive,  and  rather  limited  in  its  generalization  capabilities.  The  linear 
search  performed  by  this  process  indicates  only  a  level  of  closeness  (i.e.  pattern  variance) 
for  each  of  the  stored  reference  templates  against  the  new  input.  Recognition  problems 
are  encountered  given  slight  deviations  in  the  input  pattern,  these  deviations  most  often 
incurring  a  large  increase  in  the  computed  pattern  variance. 

HNeT,  by  comparison,  performs  a  process  that  is  similar  in  function,  but  not  in 
origin  to  the  pattern  variance  calculation.  Instead  of  separately  storing  the  reference 
template  as  ANN  does,  they  are  enfolded  on  to  the  same  space.  To  explain  enfolding,  it  is 
simply  a  process  of  superimposing  large  numbers  of  stimulus-response  patterns  upon  the 
same  set  of  scalars.  Equation  (12)  best  explains  the  mathematical  process  of  this 
phenomenon.  New  stimulus-response  patterns  are  generated,  but  instead  of  being  placed 
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in  a  different  storage  location,  they  are  placed  back  over  the  previously  computed  scalars. 
This  reduces  the  storage  requirements  for  all  of  the  reference  patterns  to  that  required  to 
store  only  one  reference  template.  For  the  HNeT  cell,  storage  requirements  are  decreased 
over  the  linear  search  method  in  direct  proportion  to  the  number  of  stimulus-response 
patterns  learned.  The  time  requirements  to  perform  a  response  recall  operation  for  an 
HNeT  neural  cell  is  also  reduced  relative  to  the  proportion  of  the  number  of  stimulus- 
response  patterns  learned. 

Within  the  HNeT  process,  analog  stimulus-response  patterns  are  most  often 
presented  to  the  neuron  cell  as  sets  of  complex  scalars  (values).  Each  complex  scalar  has 
a  phase  angle  e'e‘  which  may  represent  some  form  of  measurement  (i.e.  temperature, 
pressure,  etc.)  and  magnitude  Xj  which  represents  a  degree  of  confidence  in  that 
measurement.  This  confidence  (magnitude)  regulates  the  influence  of  the  input  signal 
during  both  learning  and  recall  operations.  The  confidence  component  may  also  exist 
within  the  generated  response  recall  signal,  in  this  case  providing  a  measure  of 
recognition  that  the  cell  has  for  the  stimulus  signal  received,  that  is  to  say  that  the  new 
(recycled)  stimulus  S*  will  be  regulated  within  that  cell  as  well. 

2.  Representation  of  Mathematical  Information 

Current  theories  in  computational  neural  dynamics  follow  from  an  idea  known  as 
the  Hebb  hypothesis.  Kartalopoulos  [Ref.  5]  says  that  in  1949,  Donald  Hebb  stated  that 
when  an  axon  of  cell  A  is  near  enough  to  excite  a  cell  B  and  repeatedly  or  persistently 
takes  place  in  firing  it,  some  growth  process  or  metabolic  change  takes  place  in  one  or 


both  cells  such  that  A’s  efficiency,  as  one  of  the  cells  firing  B,  is  increased.  Thus,  the 
synaptic  strength  (known  as  weight  w)  between  cell  A  and  cell  B  is  modified  according  to 
the  degree  of  correlated  activity  between  input  and  output.  This  type  of  learning  is  called 
Hebbian  learning.  Most  recent  neural  theory  has  expressed  this  in  great  detail  leading  to 
the  current  abundance  of  gradient-descent  type  algorithms,  whose  core  aspects  are  largely 
built  upon  linear  representations  or  real-valued  inner  product. 

As  previously  mentioned,  holographic/quantum  model  is  different  in  that  it  uses 
both  a  complex  scalar  and  nonlinear  representation  for  stimulus  information.  At  the  most 
basic  level,  one  element  of  information  within  the  HNeT  model  is  represented  by  a 
complex  scalar.  Complex  scalars  are  a  superset  of  real  scalars,  which  is  an  indication  that 
HNeT  covers  a  wider  range  of  numbers  and  has  a  computational  advantage  over  more 
conventional  neural  networks.  These  scalars  operate  with  two  degrees  of  freedom 
represented  by  phase  angle  and  magnitude.  Illustrated  in  Figure  8  is  information 
represented  by  the  complex  scalar  where  (0)  represents  information  pertaining  to  phase 
orientation,  and  ( X )  represents  magnitude  or  the  confidence  one  has  in  that  information 
(which  can  also  be  referred  to  as  weighting).  The  magnitude  component  typically  will 
vary  between  zero  and  one,  although  they  may  practicably  extend  over  any  range. 
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Figure  8.  Illustration  of  the  Information  Element.  From  Ref.[4  ]. 


There  are  some  aspects  of  the  HNeT  process  that  are  similar  to  conventional 
processes  in  respects  to  intracellular-transmission  of  signals  between  biological  neuron 
cells  which  give  support  to  complex  scalar  representation.  In  Figure  9  (representation  of  a 
cortical  cell)  a  large  amount  of  input  lines  receive  pulse  modulated  signals  in  which 
another  responding  pulse  modulated  signal  delivers  the  cell’s  response  output  in  a  manner 
analogous  to  the  axonal  process.  Although  these  signals  may  be  interpreted  in  a  variety 
of  ways,  the  more  predominant  interpretation  is  that  pulse  modulated  signals  are 
translatable  into  real- valued  scalar  quantities. 

The  actual  signal  transmission  characteristics,  illustrated  in  Figure  9  suggest  an 
equally  feasible  view  point  which  is  that  pulsed  signals  could  transmit  complex  scalar 
quantities  via  single  line  transmission.  Frequency  pulse  modulation  and  amplitude 
modulation  of  a  wave  form  is  one  direct  means  of  analog  transmission  of  complex  scalars 
along  a  single  of  line  transmission  where  frequency  modulation  could  be  interpreted  as 
phase  orientation,  and  amplitude  modulation  interpreted  as  the  magnitude  component  of 
the  complex  scalar.  However,  more  important  are  the  operational  features  observed 


22 


Figure  9.  Block  Diagram  of  a  Cortical  Cell.  From  Ref.  [4  ]. 


within  single  cell  structures  when  information  is  both  represented  and  transmitted  as 
complex  scalars  as  well  as  other  aspects  of  the  HNeT  neural  process. 

The  following  form  is  the  basic  unit  of  information  within  the  HNeT  system: 

Xeie  (13) 

Before  moving  on,  a  point  to  make  is  that  in  cases  where  Fourier  conversion  is  applied  as 
a  preprocessing  operation,  the  phase  angle  orientation  represents  the  phase  shift  for  that 
frequency  harmonic,  and  the  magnitude  represents  the  power  of  the  harmonic.  Similarly, 
low  power  stimulus  inputs  have  less  influence  over  the  stimulus-response  mappings 
learned  by  the  cell,  as  well  as  less  influence  in  the  generation  of  the  response  signal 
during  recall. 

For  the  following  discussion,  let’s  assume  that  all  stimulus  and  response  scalars 
have  unit  magnitude  for  simplification.  Phase  angle  differences  are  the  source  for 
building  up  the  memory  generated  within  the  cell  for  such  stimulus-response  associations. 

For  instance,  one  element  of  a  stimulus  may  be  represented  by  phase  orientation  e  ‘ ,  and 
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the  associated  response  by  orientation  ek  .  The  quantity  generated  mathematically  for 
associating  stimulus  (5)  element  j  to  response  (r)  element  k  is  noted  as  follows: 


where: 


Sj  •  rk 

Sj  =  V 


id: 


h=Yke' 


(14) 


This  produces  the  phase  angle  difference  e'Mi^  where: 

@diff  =  <fa-  %  (15) 

The  quantity  e'Qdiff\s  the  representation  of  a  fundamental  “quanta”  of  information,  or 
many  measured  amounts  of  information  which  stores  the  portion  of  an  associative 
mapping  that  is  learned  by  one  cortical  memory  element  for  one  pattern,  which  in  turn, 
connects  one  element  of  the  stimulus  to  one  element  of  the  response.  A  very  important 
aspect  to  make  note  of  is  that  learning  capacity  is  primarily  based  on  the  number  of 
cortical  memory  elements  (i.e.  complex  scalars),  and  this  learning  capacity  is  directly 
proportional  to  the  number  of  cortical  memory  elements  within  the  cell.  To  clarify 
further,  one  cortical  memory  element  is  physically  represented  by  one  complex  valued 
scalar.  An  example  would  be  that  if  a  cell  has  100  memory  elements,  it  is  capable  of 
learning  and  storing  100  stimulus  response  associations. 

The  HNeT  cell  also  has  the  ability  to  respond  to  the  space  of  unknown  or 
unlearned  stimuli,  commonly  referred  to  as  the  “test  set”  in  the  Supervised  Learning 
Platform  within  the  system,  through  generation  of  low  magnitude  (or  weight/confidence) 
in  the  generated  response.  There  are  fundamental  properties  that  exist  within  the  complex 
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number  domain  that  support  the  concept  of  enfolding  of  information,  and  the 
corresponding  increase  in  the  density  of  information  within  cortical  memory  elements. 

Let  (A)  be  defined  as  any  point  in  the  complex  plane.  This  point  may  also  describe  any 
path  from  the  complex  origin  to  point  A  giving  us  the  following  mathematical  equality: 

M 

A  =  Aeie  =  (16) 

M 

Simply  put,  the  above  scalar  quantity  defines  any  path  within  the  set  of  all  possible  paths 
leading  from  the  origin  to  point  A.  Figure  10  illustrates  the  above  process.  Within  one 
cortical  memory  element,  each  of  the  component  scalars  which  define  the  path  leading  to 
point  A  represent  one  association  “quanta”  that  has  been  learned. 


Figure  10.  Multiple  Pathways  Defining  a  Complex  Scalar.  From  Ref.[  4  ]. 
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3. 


Conversion  Formats 


We  discussed  previously  that  HNeT’s  stimulus  [S]  and  response[R]  sets  are 
represented  by  vectors  comprised  of  complex  scalars.  Using  the  complex  exponential, 
these  sets  are  as  follows: 

[SM^i  M  ei&2  M  ei03  M  ei9i ,. -An  d**  }  (17) 

and 


[R]  =  {yie^1  }  (18) 

External  stimuli  and  response  actions  are  predominantly  represented  by  real  numbers, 
therefore  HNeT  is  required  to  convert  the  external  real  numbers  to  internal  complex 
values.  A  generic  representation  of  this  mapping  is  as  follows: 

Sj  Ajedi  (19) 

The  sigmoidal  conversion  is  one  manner  in  which  this  conversion  may  be  applied.  The 
following  is  a  mathematical  representation  of  the  sigmoidal  process: 


p  =  mean  of  distribution  over  input  s;  k=l  to  N 
<t  =  variance  of  distribution  of  s 
Xj  =  the  assigned  confidence  level. 

This  representation  is  very  similar  to  the  sigmoidal  process  discussed  in  the  previous 
section,  however,  there  are  some  changes  to  compensate  for  phase  orientation  and 
HNeT’s  complex  process. 
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The  sigmoidal  function  as  described  above  maps  real  valued  signals  within  an 
external  environment  to  sets  of  phase  angle  orientations  in  most  cases  having  unit 
magnitude.  These  complex  scalars  are  then  read  directly  into  the  HNeT  cell.  Figure  1 1 
illustrates  the  sigmoid  conversion  format.  In  the  illustration,  notice  that  the  real  number 
domain  extends  along  the  horizontal  axis  and  varies  from  -  co  +co .  The  phase  axis 
extends  along  the  vertical  and  is  bounded  by  0  and  2 n. 


This  particular  format  allows  one  to  assign  some  value  to  the  complex  magnitude  (Xj)  or 
confidence  level  associated  with  each  real  valued  input  signal.  This  is  done  on  the  SL 
Platform  by  the  use  of  the  Parameter  Search  Settings/Learning  Rate.  This  rate  ranges 
between  zero  and  one.  Assigning  confidence  levels  to  input  signals  gives  it  control  over 
its  level  of  influence  during  both  learning  and  recall  operations.  We  can  also  describe  this 
as  phase  angle  information  being  “weighted”  in  proportion  to  its  associated  magnitude. 
For  instance,  a  stimulus  element  with  a  magnitude  of  0.0  assigned  will  have  no  influence 
in  the  learning  or  recall  of  a  stimulus-response  pattern.  However  if  a  confidence  level  of 
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1 .0  is  assigned,  an  influence  of  at  least  equal  weighting  will  be  established  for  all  other 
values  contained  within  the  stimulus  input  array. 

The  next  conversion  we  will  discuss  is  the  histogram  based  conversion.  This 
conversion  is  used  for  data  that  is  derived  from  artificial  sources.  In  other  words,  data  is 
not  derived  from  environmental  sources  or  human  factors  thus  the  distribution  may  vary 
considerably  from  a  Gausian  or  normal  distribution.  In  order  to  achieve  a  reasonable  level 
of  symmetry,  one  may  be  required  to  build  a  custom  conversion.  Constructing  a 
distribution  histogram  will  aid  in  determining  a  measure  of  the  distribution  profile  that 
records  and  sorts  bins  appropriately  centered  about  the  mean  of  the  distribution.  Once  the 
distribution  profile  is  determined,  a  polynomial  must  be  obtained  to  arrive  at  a  best  fit  for 
the  distribution  profile.  One  fit  technique  that  is  effective  is  the  Chebychev  function  or 
polynomial  given  as  follows: 

C.(X)  =  cos(narccosx  )  (21) 


This  may  be  combined  with  trigonometric  identities  to  yield: 


C„,W=2xC.W-C..,W  (22) 

for  n  >  1 

It  is  important  to  note  that  each  polynomial  has  n  zeros  in  the  interval  [-1,1],  and  they  are 
located  at  the  points: 


X  =  cos 


;r(fc  -.5)^ 

V  «  > 


for  k-  1,....,  n 


(23) 
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The  distribution  density  histogram  must  be  selected  over  an  appropriate  range  and 
centered  about  the  points  indicated  in  (23).  The  input  range  is  from  —  °°  =>  +oo ,  scaling 
this  to  [-1,1]  via  the  arctan  function.  Once  this  is  done,  distribution  bins  are  set  for 
optimal  placement  and  data  is  place  such  that  the  distribution  format  is  symmetrically 
formed. 

The  Fourier  conversion  is  a  method  of  calculation  that  performs  a  one  to  one 
translation  between  a  time  domain  series  and  a  frequency  domain  series.  The  benefit  of 
the  Fourier  transform  is  that  it  transforms  real  valued  vectors  or  surfaces  directly  into 
complex  value  sets.  It  is  important  to  note  this  method  helps  eliminate  discontinuity 
within  problems  that  may  not  be  solved  by  the  faster  conversion  methods  previously 
mentioned.  This  conversion  allows  for  better  generalization  because  data  often  assumes  a 
more  orthogonal  (i.e.  symmetrical)  state  when  expressed  in  frequency  domain,  and  data 
reduction  is  possible  at  the  input.  The  transform  is  described  mathematically  below: 

<24> 

f=o 

The  coefficients  H(f)  are  fed  into  the  cortical  cell  at  which  time  stimulus  to  response 
mappings  are  performed  within  the  complex  domain.  When  values  are  read  back  out  from 
the  cell,  the  inverse  Fourier  transform  is  performed  given  as  such: 

(25) 
/ =o 

An  illustration  of  the  Fourier  conversion  is  provided  below. 
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Figure  12.  Fourier  Conversion  on  Data.  From  Ref.  [4  ]. 


4.  Generalization 

Within  the  HNeT  model,  generalization  refers  to  aspects  concerning  the  topology 
or  characteristics  relating  to  the  interpolation  of  trained  stimulus-response  pattern  sets 
within  an  HNeT  neural  cell.  A  basic  depiction  of  the  generalization  properties  within  the 
HNeT  system  is  that  they  form  a  nonlinear  complex  based  polynomial  in  which  the  cell 
similarly  behaves.  That  is  to  say  that  the  function  interpolates  smoothly  between  the  fit 
points  (i.e.  pattern  sets  used  to  train).  For  example,  if  we  trained  data  and  the  polynomial 
fit  is  similar  to  that  of  a  fourth  order  product  of  terms,  as  seen  in  Figure  1 3 


Figure  13.  Topology  of  Cell  of  Two  Input  Dimensions  (Fourth  Order).  From  Ref.  [4  ]. 
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(trained  on  100  patterns),  the  topology  would  be  somewhat  smoother  than  that  of  fifth 
order  terms.  For  comparison,  in  Figure  14  below  (trained  on  500  patterns) ,  is  a  topology 
created  when  using  sixth  order  terms.  One  sees  that  over  a  time  scale,  the  system  creates  a 
more  complex  topology  in  that  the  greater  slopes  and  narrower  summit  provides  for  a 


more  complex  topology  which  accommodates  a  greater  stimulus-response  pattern  storage 
density. 


Figure  14.  Topology  of  Cell  of  Two  Input  Dimensions  (Sixth  Order).  From  Ref.  [4  ]. 
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III.  SIMPLE  FUNCTION  EVALUATION 

Now  that  a  general  description  of  the  HNeT  process  has  been  given,  we  will 
begin  to  investigate  its  properties  on  simple  mathematical  functions.  Random  numbers 
will  be  generated  and  used  as  inputs  to  each  function.  The  random  inputs,  which  we  will 
call  Xj  (stimulus),  and  data  produced  by  these  functions  using  the  stimulus,  X2 
(response),  will  be  applied  to  HNeT  (e.g.  for/(x)  =  cos  (x),  we  have  X2  =  cos  (Xi )).  The 
error  produced  by  the  system  will  help  to  evaluate  some  of  the  properties  of  HNeT. 

A.  EVALUATING  TRIGONOMETRIC  FUNCTIONS 

1.  Cosine  and  Cos(l/x) 

The  first  set  of  tests  were  conducted  on  various  trigonometric  functions  to 
establish  whether  HNeT  could  leam  to  emulate  the  most  basic  functions  in  mathematics. 
This  is  necessary  because  most  technological  applications  require  the  use  of  various 
functions  such  as  cosine,  tangent  and  many  others. 

The  cosine  function  was  first  function  that  was  tested.  This  function  was  tested 
using  300  values  of  generated  random  input  data,  Xj,  and  the  corresponding  outputs  X2  = 
cos  (Xj  ).  For  the  purpose  of  our  test,  we  will  split  this  data  into  training  and  testing  sets 
with  80%  in  the  training  set  and  20%  in  the  testing  set  (240  and  60  records  respectively). 
This  test  was  conducted  with  4  epochs  (training  exposures  per  cycle)  and  a  learning  rate 
(magnitude/confidence)  of  .25.  The  results  of  the  test  yield  a  mean  absolute  error  of  .0061 


33 


for  the  training  set  and  .0075  for  the  test  set.  This  was  found  in  5  cycles  using  777 
memory  elements  out  of  a  total  1 000.  This  is  expected  because  the  training  set  has  more 
data  available  so  that  HNeT  can  leam  to  recognize  more  of  the  presented  cases.  The  test 
set,  however,  uses  fewer  cases  but  still  has  a  reasonable  error,  which  is  a  good  indication. 

If  we  continue  to  observe  the  results,  we  will  find  that  as  the  parameter  search 
continues  to  cycle,  the  error  for  the  test  and  training  results  increase.  This  is  indicated  by 
the  error  graph  in  Figure  15.  The  results  show  that  at  the  85th  cycle,  the  test  error  (error 
#1)  is  .1091,  and  the  training  error  (error  #2)  is  .1157.  This  was  done  in  23  memory 


Figure  15.  Cosine  Error  Graph  80/20. 


elements  out  of  1 000.  The  explanation  for  this  is  that  as  HNeT  cycles  through  the 
data,  the  process  performs  a  function  analogous  to  the  pattern  variance  calculation, 
however,  the  reference  templates  are  enfolded  on  to  the  same  storage  space.  As  the 
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storage  requirement  reduces  for  all  of  the  reference  patterns  to  only  that  amount  of 
memory  required  to  store  a  single  reference  template,  the  average  error  on  response  recall 
gradually  increases. 

A  second  evaluation  was  performed  on  the  same  set  of  data,  this  time  splitting  the 
data  into  training  and  test  sets  of  equal  size  (150  records  each).  This  time,  the  results 
yielded  a  mean  absolute  error  of  .0290  for  the  test  set  and  .0257  for  the  training  set.  This 
was  executed  in  16  cycles  using  only  470  memory  elements  out  of  1000.  The  resulting 
test  set  error  was  much  closer  to  the  training  set  error.  Due  to  increased  data  of  the  test 
set,  its  pattern  recognition  capability  increased.  Overall,  both  sets  of  error  increased 
which  is  believed  to  be  caused  by  a  decrease  in  training  records  evaluated  for  the  training 
set,  and  in  the  test  set,  HNeT  has  incorrectly  used  data  as  the  enfolding  process  took 
place  thereby  limiting  its  generalizing  capability.  The  fact  that  fewer  memory  elements 
were  used  limited  generalizing  capabilities  causing  increased  error  for  both  sets.  As 
HNeT  continued  to  run  through  its  parameter  search,  its  minimum  memory  elements  were 
20,  yielding  an  error  of  .1 148  for  the  test  set  and  .1153  for  the  training  set  at  the  90th 
cycle.  This  is  similar  to  the  outcome  in  the  previous  test,  which  shows  for  these  two 
evaluations  for  this  particular  function,  there  is  a  tendency  to  converge  to  the  same  error 
near  the  same  cycle.  Figure  16  captures  the  error  data  for  the  second  test  in  graph  form. 

We  will  next  introduce  the  trigonometric  function,  Cos(l/x),  for  further 
investigation  of  the  properties  of  HNeT.  This  function  is  important  to  test  because  of  its 
unbounbed  variation.  That  is,  as  x  approaches  zero,  the  oscillations  become  extremely 


35 


rapid,  indeed  the  function  oscillates  between  minus  one  and  one  an  infinite  number  of 
times  in  any  open  interval  (0,  £)  for  any  £  >0.  When  evaluating  this  function,  we  took  a 
similar  approach  to  that  of  the  cos(x)  function.  Initially,  we  started  with  200  data  values 
with  80%  of  the  data  to  be  evaluated  as  the  training  set  and  20%  of  the  data  for  the  tested 
set.  As  before,  we  assigned  Xi  (the  generated  random  input  value)  as  the  stimulus  and  X2 
(the  corresponding  output  value)  as  the  response.  This  resulted  in  a  significant  increase  in 
error.  The  test  set  data  resulted  in  an  error  of  .4587,  and  the  training  set  resulted  in  an 
error  of  .0863  in  the  9th  cycle  utilizing  635  memory  elements  out  of  a  possible  1000. 
Considering  the  range  of  the  numerical  value  of  the  data,  this  error  is  considerably  large. 
The  data  for  this  test  set  range  from  roughly  -2.0  to  +2.0  for  the  stimulus  value  and  -1.0 
to  +1.0  for  the 
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Figure  16.  Cosine  Error  Graph  50/50. 
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response.  In  an  attempt  to  decrease  the  error,  we  increased  the  number  of  epochs  (the 
number  of  training  exposures  in  one  cycle)  from  four  to  15  and  the  learning  rate 
(magnitude/confidence)  remained  the  same  at  .25.  There  was  a  slight  decrease  in  both 
errors.  The  decrease  was  due  to  increased  training  exposures,  giving  HNeT  more  chances 
to  leam  the  data.  The  training  set  error  result  was  .0693,  and  the  test  set  error  was  .4058 
in  the  6th  cycle,  however,  the  result  is  still  rather  large.  HNeT  is  expected  to  have 
difficulty  with  this  function  due  to  its  unbounded  variation.  The  system  can  not  keep  up 
with  the  rapid  fluctuations  of  this  function.  The  error  graph  in  Figure  17  gives  visual 
evidence  of  HNeT’ s  difficulty  in  learning  this  data,  as  shown  by  the  test  set  error. 

To  determine  if  the  tests  of  the  function  are  valid  for  both  the  training  set  and 
the  test  set,  there  is  a  procedure  in  HNeT  known  as  validation.  In  this  procedure,  HNeT 
learns  to  emulate  the  entire  set  of  data,  and  generates  the  mean  absolute  error  for  the 
entire  data  set  as  a  whole.  It  also  gives  a  pictorial  view  of  the  function  graph  on  the 
Supervised  Learning  Platform  as  a  whole  verses  individually  by  sets.  The  validation 
procedure  was  conducted  for  the  previous  test  and  the  resulting  combined  mean  absolute 
error  was  .1369.  We  will  continue  to  investigate  the  properties  of  HNeT  using  another 
trigonometric  function  to  determine  its  capabilities  relative  to  discontinuity. 
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2.  Tangent 

The  tangent  function  unlike  the  cosine  function  is  one  that  is  not  defined  for  all  x. 
Since  tanx  =  sinx/ cos  x,  the  tangent  function  is  undefined  whenever  cos  x  =  0;  this 
occurs  when  x  is  an  odd  integer  multiple  of  7t  /2  ,  that  is  when 


Table  1  shows  some  values  generated  by  the  tangent  function  from  randomly  generated 
numbers.  Notice  that  as  x  approaches  n/2  =  1 .571,  tan  x  starts  to  increase  sharply.  This  is 
because  sin  x  is  approaching  one,  while  cos  x  is  approaching  zero.  From  this  data  we  will 
investigate  how  HNeT  responds  with  the  sharp  increases  near  the  singularities  ±n!  2 . 
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X 

Tan  x 

1.191 

2.504 

1.254 

3.050 

1.290 

3.470 

1.339 

4.247 

1.415 

6.372 

1.443 

7.813 

1.472 

10.13 

1.478 

10.85 

1.488 

12.12 

1.535 

28.04 

1.553 

56.96 

1.592 

-45.15 

1.623 

-18.93 

1.692 

-8.180 

1.864 

-3.305 

1.957 

-2.456 

2.112 

-1.663 

2.183 

-1.423 

Table  1.  Tangent  Values  for*. 


For  the  testing  of  the  tangent  function,  we  generated  200  data  values  of  Xi 
(stimulus)  and  X2  (response).  We  will  use  80%  of  the  values  for  the  training  set  and  20% 
of  the  values  for  the  test  set.  From  the  first  test,  using  the  default  setting  of  4  epochs  and 
learning  rate  of  .25,  the  training  set  error  was  1.424  and  the  test  set  error  was  5.682  in  the 
15th  cycle  with  904  memory  elements  used.  In  an  attempt  to  decrease  error,  the  number  of 
epochs  and  learning  rate  were  adjusted  until  the  best  results  were  found.  After  numerous 
trials,  the  error  remained  relatively  unchanged,  however,  the  memory  elements  decreased 
almost  two  fold.  By  increasing  the  epochs  to  15  and  slowing  the  learning  rate  to  .01, 
HNeT  was  able  to  use  less  memory  to  accomplish  these  results.  The  error  for  the 
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training  set  was  1.619  and  the  test  set  was  5.695  in  the  15th  cycle,  using  494  memory 
elements.  The  following  table  is  an  example  of  the  results  near  the  singularities.  Notice 
that  HNeT  didn’t  respond  very  well  near  the  singularities  because  of  the  sharp 


Xi  tan  x  HNeT  Error 


1.415 
1.443 
1.472 
1.478 
1.488 
1.535 
1.553 
1.592 
1.623 
1.692 
1.864 
1.957 
2.112 
2. 183... 


6.372 

7.813 

10.13 

10.85 

12.12 

28.04 

56.96 

-45.15 

-18.93 

-8.180 

-3.305 

-2.456 

-1.663 


8.319 

8.617 

8.852 

8.894 

8.949 

9.084 

9.074 

8.931 

8.711 

7.912 

4.838 

3.052 

0.455 


-1.947 

-0.803 

1.285 

1.958 

3.172 

18.95 

47.88 

-54.08 

-27.64 

-16.09 

-8.144 

-5.508 

-2.118 

-0.899 


-1 .423  -0.524 


Table  2.  HNeT  Results  of  the  Tangent  Function. 


increase  in  the  values.  Even  as  a  result  of  the  decrease  in  the  learning  rate,  HNeT  still 
produced  high  error.  The  pattern  for  this  function  is  one  with  predominantly  low  values 
for  each  stimulus.  As  HNeT  recognizes  the  pattern,  it  was  not  able  to  adjust  to  the  sharp 
increases  in  the  values.  Further  examination  is  required  to  adjust  for  the  sharp  increases 
of  this  function. 

For  the  purpose  of  decreasing  the  error  in  HNeT,  we  will  treat  the  values  near  the 
singularities  as  outliers  and  eliminate  them.  After  eliminating  values,  the  remaining  data 
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gives  172  stimulus  and  response  values  to  test.  Using  the  same  80/20  ratio  as  before,  the 
results  of  the  evaluation  yielded  an  error  of  .0433  for  the  training  set  and  .5844  for  the  test 
set  in  the  26th  cycle  utilizing  271  memory  elements.  The  number  of  epochs  remained  the 
same,  however,  HNeT  responded  better  with  an  increase  in  the  learning  rate  to  .50. 


Desired  —  HNeT 


Figure  18.  Validation  Function  Graph  With  Outliers  Eliminated. 

When  adjusting  the  ratio  to  50/50  and  increasing  the  epochs  to  30,  HNeT  decreased  its 
error  even  further  with  a  training  set  error  of  .0304  and  test  set  error  of  .3878  in  the  17th 
cycle,  utilizing  only  1 92  memory  elements.  The  validation  for  this  data  has  an  error  of 
.2091.  Below  is  the  function  graph  of  the  data  with  outliers  removed.  Notice,  the  right  tail 
of  the  function  shows  that  HNeT  (test  set)  doesn’t  respond  well  as  the  data  increases, 
which  is  the  cause  for  the  majority  of  the  error. 

There  are  other  functions  that  we  will  investigate  however,  there  is  some  limited 
evidence  that  HNeT  has  problems  in  reproducing  functions  with  sharp  increases  or 
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decreases  in  data  that  produces  sharp  curves.  In  the  following  evaluations,  we  will  give 
examples  of  HNeT’s  capability  in  evaluating  functions  whose  data  produce  curves  that 
are  smoother  and  more  gradual  (up  or  down). 

B.  EVALUATING  NATURAL  LOGARITHMIC,  EXPONENTIAL,  AND 
POLYNOMIAL  FUNCTIONS 

1.  Natural  Logarithm 

The  natural  log  function  (Inx )  is  one  that  can  be  described  as  a  function  that  has  a 
smooth  curve  with  a  gradual  increase  as  x  increases.  Some  properties  to  note  about  this 
function  are  that: 

lnx>0  if  x  >  1 

lnx  <0  if  0  <  x  <  1 

lnx  =  0  if  x  =  1 . 

Furthermore,  this  function  is  undefined  for  x  <  0. 

For  the  purpose  of  our  evaluation  of  HNeT,  we  will  use  300  data  values.  The 
results  of  the  test  produced  very  low  error  for  this  function.  The  training  set  error  was 
.0052  and  the  test  set  error  was  .0071  in  the  4th  cycle.  These  results  were  accomplished 
with  15  epochs  using  817  memory  elements.  The  error  graph  (Figure  19) 
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Figure  19.  Natural  Log  Function  Error  Graph. 


shows  how  well  the  test  set  remains  close  to  the  training  set  despite  the  lower  amounts  of 
data  (60  out  of  300)  to  evaluate.  HNeT  responds  very  well  to  the  log  function.  However, 
we  will  continue  to  examine  other  functions  to  further  investigate  its  properties. 


2.  Exponential  Function 

The  exponential  function  (ex)  is  another  function  that  produces  a  smooth  curve. 
The  exponential  function  is  continuous  for  all  values  ofx.  In  our  case,  x  >  0  and  has  the 
following  properties: 


ex  >  0  for  all  x 

limit  ex  as  x— >  +  oo  =  +  oo 

limit  ex  as  x— »  -  oo  =  0 . 
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The  function  was  also  tested  using  300  data  values  with  the  same  format  as  the  natural 
log  function.  Again,  the  results  of  this  function  produced  very  low  error  for  both  sets  of 
data.  During  this  particular  evaluation,  the  memory  elements  were  considerably  lower 
than  those  of  the  log  function.  The  memory  elements  used  were  384  out  of  1000  total. 
This  was  accomplished  with  15  epochs.  The  error  was  .0013  for  the  training  set  and 
.0014  for  the  test  set  in  the  19th  cycle.  Numerous  evaluations  for  this  function  were 
conducted  with  various  epoch  changes  as  well  as  learning  rate  changes.  For  each 
evaluation,  the  error  produced  was  relatively  low  with  the  currently  mentioned  evaluation 
having  the  lowest  error. 

As  HNeT  tried  to  emulate  this  function  and  the  enfolding  process  took  place,  the 
error  for  the  test  set  began  to  grow  at  a  slower  rate  than  that  of  the  training  set  around  the 
27th  cycle  indicating  that  the  test  set  was  doing  a  better  job  of  reproducing  the  function 
than  the  training  set.  This  trend  is  shown  in  Figure  20. 

HNeT  continues  to  show  that  for  functions  with  smooth  curves  with  gradually 
increasing  data,  it  performs  rather  well.  The  next  function  to  be  introduced  to  the  system 
is  the  polynomial  function.  With  this  function,  we  will  continue  to  examine  HNeT’s 
properties. 
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Figure  20.  Error  Graph  for  the  Exponential  Function. 


3.  Polynomial 

Given /(x)  =  x3  +  4x2  -  lx  -10 ,  this  randomly  selected  polynomial  is  a  function 
that  is  continuous  for  all  x.  In  other  words,  the  domain  of  the  polynomial  is  (— co,+co) . 
This  polynomial  will  be  evaluated  in  the  HNeT  system  to  examine  HNeT’s  capability 
with  this  type  of  function. 

The  first  tests  was  conducted  with  300  data  values,  again  using  the  same  format  as 
the  exponential  function.  We  began  with  the  default  setting  of  4  epochs  and  a  learning 
rate  of  .25.  The  results  for  this  evaluation  yielded  an  error  of  .0059  for  the  test  set  and 
.0073  for  the  training  set  in  the  10th  cycle.  The  memory  elements  used  were  285  out  of 
1000.  Continuing  on  with  more  evaluations,  the  best  results  yielded  a  test  set  error  of 
.0018  and  a  training  set  error  of  .0026  in  the  12th  cycle.  This  was  accomplished  with  15 
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epochs  and  546  memory  elements.  For  this  evaluation  the  test  set  error  was  lower  than  the 
training  set  for  all  evaluations  conducted.  The  error  graph  for  this  function  was  very 
similar  to  the  error  graph  of  the  exponential  function.  This  is  an  indication  of  how  well 
HNeT  responds  to  these  functions.  It  responded  to  untrained  data  better  indicating  that 
HNeT  was  able  to  easily  leam  and  recognize  smooth  and  gradually  increasing  data  . 

In  the  next  section,  HNeT  will  be  examined  on  truly  random  data.  That  is,  there 
will  be  no  associated  function  to  determine  a  particular  pattern  to  follow. 
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IV.  RANDOM  DATA  EVALUATION 


The  California  Lottery  began  on  October  14,  1986.  In  its  beginning,  the  draw  was 
based  on  picking  six  numbers  from  1  to  49.  However,  the  numbers  increased  from  49  to 
53  with  six  picks  on  June  21,  1990.  On  December  15, 1991,  the  draw  was  changed  and  is 
known  as  the  “Super  Lotto”.  To  this  current  day,  the  numbers  drawn  range  from  1  to  51 
with  six  picks. 

The  first  set  of  tests  are  based  on  the  collection  of  California  Lottery  results.  This 
is  a  truly  random  data  set  that  is  publicly  available  via  the  Internet.  It  is  an  ordered 
collection  of  the  results  of  every  draw  of  the  California  lottery  from  October  14,  1986  up 
to  the  current  drawing.  Each  record  consists  of  6  ordered  distinct  integers  whose  values 
range  from  1  to  5 1 .  Since  the  drawing  is  conducted  physically  (with  numbered  balls),  the 
data  are  truly  random  representatives  of  their  particular  distribution.  This  is  crucial  for 
testing  neural  network  software  since  pseudo-random  sequences  often  have  underlying 
structure  that  can  produce  statistically  abnormal  results  in  such  tests.  Initial  test  were 
conducted  with  all  drawings  dating  back  to  October  14, 1986,  however,  since  there  were 
differences  in  the  range  of  numbers  drawn,  the  results  were  erroneous.  For  the  purpose  of 
our  current  tests,  we  trimmed  the  data  set  to  those  dating  back  to  December  15,  1991.  We 
will  use  data  from  868  recent  draws  that  will  be  split  into  training  and  testing  sets  of 
equal  size  (434  records  each). 
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The  values  of  the  6  numbers  in  the  drawing  are  random  variables;  we  will  call  them 
Xi,  X2,  X3,  X4,  X5)  and  X6  respectively.  Note  that  the  knowledge  of  the  value  of  some  X( 
gives  information  that  can  be  used  to  narrow  down  the  possible  values  of  the  others.  For 
instance,  if  we  are  given  that  X5  =  40  then  we  can  infer  that  X6  must  take  on  a  value 
between  41  and  51  with  each  possible  value  being  equally  likely.  This  is  simply  a 
conditional  probability,  in  particular  Pr{  X6  =  x  I  X5  =  y}  =  1/(5 1-y)  if  x>y  and  zero 
otherwise. 

In  the  first  test,  we  attempt  to  train  HNeT  to  guess  Xg  given  the  value  of  X5.  This  is 
an  interesting  test  probabilistically  since  Pr{  X6 1  Xj,  X2,  X3,  X4,  X5}  =  Pr{  X$  I  X5},  that 
is,  X$  is  conditionally  independent  of  Xi  through  X4  if  you  are  given  X5  (this  is  similar  to 
a  Markov  property  in  a  stochastic  process). 

The  best  linear  unbiased  estimate  (BLUE)  of  X$  given  that  X5  =  y  is  X  =  (52  + 
y)/2.  In  Table  3,  we  show  the  X*  as  well  as  the  estimate  generated  by  HNeT,  denoted  XH, 
for  all  possible  values  of  X5.  Note  that  in  this  table  HNeT  was  trained  using  only  the 
values  of  X5  and  X6.  From  the  data,  it  is  apparent  that  the  response  generated  by  HNeT 
attempts  to  approach  the  optimal  response  although  it  is  far  from  it  in  some  areas.  The 
mean  absolute  error  over  the  training  set  was  3.320  and  3.61 1  over  the  test  set.  HNeT 
found  this  solution  in  one  cycle  using  1 00  memory  elements  out  of  a  possible  1 00. 

Another  test  was  conducted  using  X4  as  the  stimulus  and  X6  as  the  response.  The 
results  of  this  test  yielded  a  higher  error  value  for  both  sets.  The  mean  absolute  error  for 
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X5_ 

X* 

XH 

Error 

5 

28.50 

34.37 

-5.87 

6 

29.00 

30.12 

-1.12 

7 

29.50 

30.43 

-0.93 

8 

30.00 

36.27 

-6.27 

9 

30.50 

29.01 

1.49 

10 

31.00 

31.88 

-0.88 

11 

31.50 

31.05 

0.45 

12 

32.00 

35.95 

-3.95 

13 

32.50 

34.23 

-1.73 

14 

33.00 

34.28 

-1.28 

15 

33.50 

33.55 

-0.05 

16 

34.00 

33.51 

0.49 

17 

34.50 

36.04 

-1.54 

18 

35.00 

36.33 

-1.33 

19 

35.50 

37.10 

-1.60 

20 

36.00 

44.44 

-8.44 

21 

36.50 

36.76 

-0.26 

22 

37.00 

34.12 

2.88 

23 

37.50 

36.74 

0.76 

24 

38.00 

37.36 

0.64 

25 

38.50 

38.71 

-0.21 

26 

39.00 

37.27 

1.73 

27 

39.50 

37.59 

1.91 

28 

40.00 

32.70 

7.30 

29 

40.50 

40.61 

-0.11 

30 

41.00 

40.44 

0.56 

31 

41.50 

41.33 

0.17 

32 

42.00 

42.53 

-0.53 

33 

42.50 

42.28 

0.22 

34 

43.00 

40.71 

2.29 

35 

43.50 

45.15 

-1.65 

36 

44.00 

43.80 

0.20 

37 

44.50 

39.19 

5.31 

38 

45.00 

43.70 

1.30 

39 

45.50 

39.77 

5.73 

40 

46.00 

45.49 

0.51 

41 

46.50 

43.93 

2.57 

42 

47.00 

44.60 

2.40 

43 

47.50 

47.61 

-0.11 

44 

48.00 

45.16 

2.84 

45 

48.50 

40.63 

7.87 

46 

49.00 

48.20 

0.80 

47 

49.50 

41.59 

7.91 

48 

50.00  . 

49.67 

0.33 

49 

50.50 

49.44 

1.06 

50 

51.00 

51.31 

-0.31 

Table  3.  BLUE  Comparison.  After  Ref.  [6]. 


the  training  set  is  4.060  and  4.353  for  the  test  set.  However,  this  was  found  using  24 
memory  elements  in  37  cycles,  much  less  memory  than  the  previous  test.  The  error  was 
expected  to  be  higher  because  as  stated  before,  if  we  are  given  that  X4  =  30  and  X5  =  40, 
X$  must  take  on  a  value  between  41  and  51  with  each  possible  value  being  equally  likely, 
rendering  X4  statistically  unimportant. 

For  the  third  part  of  this  test  we  let  HNeT  train  using  the  full  data.  That  is,  attempt 
to  guess  the  value  ofX*  given  the  values  of  Xi,  X2,  X3,  X4,  and  X5  -  When  presented  with 
this  new  task  we  get  very  interesting  results.  The  resulting  artificial  neural  network 
(ANN)  had  a  mean  absolute  error  over  the  training  set  of  only  3.128  but  3.836  over  the 
test  set.  This  was  found  in  6  cycles  and  used  77  memory  elements.  The  interesting 
phenomenon  here  is  that  the  error  improves  over  the  training  set  because  there  is  more 
data  available  so  that  HNeT  can  learn  to  recognize  more  of  the  presented  cases.  However, 
the  error  is  worse  over  the  tests  set.  This  happens  because  in  this  test  HNeT  has 
incorrectly  used  data  in  constructing  the  ANN  that  is  statistically  unimportant.  In  the 
following  graphs  we  will  discuss  the  results  based  on  generalization. 

The  first  graph  shown  (Figure  21)  is  a  representation  of  the  response  (Y  axis) 
verses  X4  (X  axis)  and  X5  (Z  axis).  Notice  that  the  gradients  are  not  as  steep  at  most 
points  but  steep  for  some.  This  is  a  representation  of  HNeT  responding  well  to  most 
stimuli  because  X5  and  X4  have  data  that  is  relatively  close  in  value  to  the  response  data 
X6.  However,  there  are  also  steep  gradients  that  are  caused  by  low  values  in  X,  and  X5. 
For  the  most  part,  the  graph  shows  that  the  stimuli  interpolates  much  smoother  than  that 
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of  the  lower  values  Xj.  The  second  graph  (Figure  22)  shows  the  response  verses  Xj  (Z 
axis)  and  X5  (X  axis).  Since  most  values  in  Xj  represent  values  between  one  and  ten,  and 
Pr{  X<;  |  Xi,  X2,  X3,  X4,  X5}  =  Pr{  X6 1  X5),  Xi  has  little  or  no  effect  on  the  outcome  of 
the  response.  This  is  evident  in  the  graph  that  shows  the  Z  axis  being  relatively  flat, 
whereas  the  X  axis  has  a  much  more  linear  fit. 


Figure  21 .  Topology  of  Cell  Input  X4  and  X5. 

The  test  conducted  with  Xi  as  the  stimulus  and  X^  as  the  response  backs  up  the 
graphical  evidence  that  Xi  is  statistically  unimportant.  The  test  set  error  was  4.450,  and 
the  training  set  error  was  4.810  using  72  memory  elements  in  7  cycles  which  gives  further 
evidence  of  its  unimportance. 
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Figure  22.  Topology  of  Cell  Input  Xi  and  X5. 
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V. 


FINDINGS  AND  CONCLUSIONS 


A.  FINDINGS 

The  HNeT  system  is  an  artificial  neural  network  that  has  shown  considerable 
evidence  of  being  a  more  reliable,  efficient,  and  effective  tool  than  some  of  the  more 
traditional  artificial  neural  networks.  This  is  evident  by  its  performance  in  evaluating 
simple  functions  and  random  data.  Not  only  was  it  fairly  accurate  in  its  evaluations,  it 
provided  many  other  means  of  determining  the  accuracy  of  its  outcomes;  for  instance, 
error  graphs,  topology  graphs,  function  graphs,  and  validation  capabilities.  HNeT  is  also  a 
very  user  friendly  system.  The  ability  to  easily  change  settings  within  each  evaluation  will 
reduce  time  constraints  for  finding  solutions  to  research  problems. 

Its  ability  to  leam  many  stimulus-response  patterns  with  a  single  cell  comprised  of 
complex  scalars  decreases  its  storage  capacity  requirements  which  allows  for  a  decrease 
in  the  time  to  perform  a  response  recall  operation.  This  capability  will  allow  for  increased 
volumes  of  problem  solving  and  evaluating. 

Although  in  most  cases  the  error  increased  as  the  memory  elements  decreased, 
HNeT  had  already  found  its  best  case  evaluation,  usually  in  an  early  cycle.  This  shows  its 
capability  responding  to  stimuli  accurately  and  early. 
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B.  CONCLUSIONS 


HNeT  is  a  system  that  provides  a  valuable  means  for  problem  solving.  Although 
the  results  from  the  evaluations  conducted  in  this  thesis  were  favorable,  other  evaluations 
should  be  conducted  to  further  examine  this  system’s  capabilities.  Image  processing  is  an 
application  that  is  well  worthy  of  consideration  for  future  evaluations.  This  will  provide 
great  insight  into  its  true  capabilities  as  it  relates  to  military  applications. 

Most  military  doctrine,  whether  friendly  or  enemy,  is  established  based  on  some 
repetitious  pattern  of  war.  In  other  words,  our  enemy’s  pattern  of  warfare  dictate  how  we 
fight.  From  these  patterns,  and  our  previous  evaluations,  HNeT  has  the  capability  to  learn 
and  provide  information  based  on  these  previous  patterns. 
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